fix(validate): catch workspace setup_error causes at validate-time#1087
Merged
fix(validate): catch workspace setup_error causes at validate-time#1087
Conversation
Adds validateWorkspacePaths() to detect two classes of errors that surface as setup_error at runtime but are statically checkable: 1. workspace.hooks.*.command — script arguments that look like relative file paths (./foo, ../bar/setup.mjs) are resolved using the same cwd precedence as the runtime (hook.cwd ?? workspaceFileDir ?? evalDir) and checked for existence. 2. workspace.template — the template path is resolved and checked. Both checks apply to inline workspace configs and to paths inside external workspace files (workspace: "path.yaml"). The external file existence case is already caught by eval-validator; the new validator focuses on the internal hook script paths and template that were previously only caught at runtime. Closes #1078 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Deploying agentv with
|
| Latest commit: |
d239a1f
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://dbe1a131.agentv.pages.dev |
| Branch Preview URL: | https://fix-1078-validate-workspace.agentv.pages.dev |
- Combine duplicate node:fs/promises imports into one - Fix JSDoc: "two classes" now correctly lists two, clarify external file note - Remove redundant workspace string existence check (eval-validator owns it) - Add command arg index to error location: command[1] instead of command - Add .bash and .zsh to script extension heuristic - Add tests for: after_each/after_all hooks, script alias, external workspace template Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Collaborator
Author
Post-review fixes appliedAddressed code review findings:
Regarding the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1078
Problem
agentv validatepassed eval files that would fail at runtime withsetup_errorwhen workspace hook scripts or template paths didn't exist. These errors are detectable statically.Changes
New
validateWorkspacePaths()function inpackages/core/src/evaluation/validation/workspace-path-validator.ts. Called alongsidevalidateFileReferencesfor all eval files.What it catches
1. Hook script paths — command args that look like relative file paths (
./,../, or script extensions:.mjs,.js,.ts,.sh,.py, etc.) are resolved using the same cwd precedence as the runtime:If the resolved path doesn't exist →
error.2. workspace.template — the template path is resolved and checked for existence.
Both checks apply to inline
workspace:objects and to paths inside external workspace files (workspace: "path.yaml").What it doesn't flag
node,bash, etc.) — no path separator or script extension-c,echo hello, etc.)Red/Green UAT
RED — hook script missing:
RED — template missing:
GREEN — valid eval, no workspace:
Tests
10 new unit tests covering all cases: no workspace, valid/missing workspace file ref, valid/missing template, valid/missing hook script, system binary (not flagged), hook cwd resolution, external workspace file hooks.
🤖 Generated with Claude Code